Enabling Large Batch Size Training for DNN Models Beyond the Memory Limit While Maintaining Performance

Recent deep learning models are difficult to train using a large batch size, because commodity machines may not have enough memory accommodate both the model and data size. The size is one of hyper-parameters used in training model, it dependent on limited by target machine capacity can only fit into remaining after uploaded. Moreover, item also an important factor if each larger then that becomes smaller. This paper proposes method called Micro-Batch Processing (MBP) address this problem. helps providing processing splits processes them sequentially. After small batches individually, loss normalization algorithm based gradient accumulation maintain performance. purpose our allow sizes exceed system without increasing or multiple devices (GPUs).